<jupyter_start><jupyter_text>AdultData (Shephertz Technologies)<jupyter_code>#importing the libraries
import numpy as np
import pandas as pd

from sklearn.model_selection import KFold 
from sklearn.model_selection import cross_val_score

from sklearn.tree import DecisionTreeClassifier
import sklearn.ensemble as ske 

import matplotlib.pyplot as plt

%matplotlib inline<jupyter_output><empty_output><jupyter_text>The Adultdata.xlsx dataset contains 13 different attributes of an individual such as his/her Workclass, Education, Marital Status, etc. and on target variable. The target variable specifies whether that individuals salary is less than or more than 50K.<jupyter_code># reading the exel file
data = pd.read_excel("C:/Users/hp/Desktop/Shephertz/Adultdata.xlsx", sheet_name = 0, header = 0)
data.head()<jupyter_output><empty_output><jupyter_text>Data PreprocessingChecking and dropping the number of null values in our dataset.<jupyter_code>print(data[data.isnull().any(axis = 1)].shape)
data = data.dropna()
print(data.shape)
data = data.replace(' ?', np.nan)
data.dropna()
data.shape<jupyter_output><empty_output><jupyter_text>Understanding the data with **decriptive statistics**.<jupyter_code>data[['Workclass', 'Education', 'Martial Status', 'Occupation', 'Relationship', 'Race', 'Sex', 'Country', 'Target']].describe()<jupyter_output><empty_output><jupyter_text>Visualizing some insightsWe tried to gain some insights by statistically visualizing the *Adultdata*. Below is the code generating different pandas series for the number of each race earning a salary greater and lower than 50K.<jupyter_code># filter for salary less than 50K
less_salary = data['Target'].str.contains('<=50K')
low_sal = data[less_salary]
print(low_sal.shape)

#filter for salary more then 50K
more_salary = data['Target'].str.contains('>50K')
high_sal = data[more_salary]
print(high_sal.shape)
# counting the number of each race of salary <=50k
race_count1 = low_sal[['Target','Race']].groupby('Race').count()
print(race_count1, '\n')

# counting the number of each race of salary >50k
race_count2 = high_sal[['Target','Race']].groupby('Race').count()
print(race_count2)<jupyter_output>Target
Race                       
 Amer-Indian-Eskimo     171
 Asian-Pac-Islander     462
 Black                 1615
 Other                  153
 White                12522 

                     Target
Race                       
 Amer-Indian-Eskimo      19
 Asian-Pac-Islander     154
 Black                  227
 Other                   13
 White                 4265<jupyter_text>Now we have the count of different Races earning salaries of lower and higher categories respectively. But, just the statistics are not enough to visualize the full picture clearly. We used matplotlib library to plot the data and gain a better understanding.<jupyter_code>values = list(race_count1["Target"])
names = list(race_count1.index)

values2 = list(race_count2["Target"])
names2 = list(race_count2.index)

plt.figure(1, figsize = (10,10))
plt.bar(names, values)
plt.bar(names, values2)

plt.title('Frequency of each race earning salary <=50K (Blue) and >50K (Orange)')
plt.xlabel('Races', size = 14)
plt.ylabel('Count', size = 14)
plt.show()<jupyter_output><empty_output><jupyter_text>It can be seen from the above bar graph that the White people are equally dominating in the number of peoples in both the salary categories. The share of Black people is far less in spite of being the largest local minority(12.1%) in the USA ( source: Wikipedia)<jupyter_code># counting the number of each sex of salary <=50k
sex_count1 = low_sal[['Target','Sex']].groupby('Sex').count()
print(sex_count1, '\n')

# counting the number of each sex of salary >50k
sex_count2 = high_sal[['Target','Sex']].groupby('Sex').count()
print(sex_count2)

values = list(sex_count1["Target"])
names = list(sex_count1.index)

values2 = list(sex_count2["Target"])
names2 = list(sex_count2.index)

plt.figure(1, figsize = (5,5))
plt.bar(names, values)
plt.bar(names, values2)

plt.title('Frequency of each sex earning salary <=50K (Blue) and >50K (Orange)')
plt.xlabel('Races', size = 14)
plt.ylabel('Count', size = 14)
plt.show()<jupyter_output>Target
Sex            
 Female    5786
 Male      9137 

         Target
Sex            
 Female     709
 Male      3969<jupyter_text>Another significant finding is the share of women in high salary working class. The number of women getting paid higher than 50k is drastically less compared to the number of women in lower salary category. Deploying Tree AlgorithmsDecision tree algorithms are deployed because they can work with both *qualitative* and *quantitative* types of variables we are having in our dataset.  Converting features into categorical variables and codingFor deploying machine learning classification algorithms it is necessary to transform the string lists into categorical variables. We can then form integer codes from those qualitative categorical variables which are better understood by the classifier.<jupyter_code>features = ['Workclass', 'Martial Status', 'Occupation', 'Relationship', 'Race', 'Sex', 'Country', 'Target']

for column in features:
    data[column] = data[column].astype("category").cat.codes<jupyter_output><empty_output><jupyter_text>Deleting unnecessary features can save both time and effort. The data for features such as **Capital Gain** and **Capital Loss** is of no use in the prediction of salary as for most of the people its value is zero.<jupyter_code>data.drop(['Education', 'Capital Gain', 'Capital Loss'], axis = 1, inplace = True)
data.head()<jupyter_output><empty_output><jupyter_text>Our predictor variables are the first 11 parameters and the target variable is the last parameters which determine whether the salary of a person is less than or more than 50K. It is necessary to create separate *nd arrays* for predictor and target variables in order to train them on different statistical learning algorithms using *sklearn* library<jupyter_code>#converting the dataset into predictor and target arrays
array = data.values
X = array[:, 0:11]
Y = array[:, 11]<jupyter_output><empty_output><jupyter_text>Determining the important featuresWe can determine the potential or importance of each feature in predicting the target variable using the *ExtraTreesClassifier* class in sklearn ensembles. Though there are many algorithms to select the best features such as *Recursive Feature Elimination* and *Principal Component Analysis* it is better to use ExtraTreesClassifier when you are planning to deploy Decision Tree Classification.<jupyter_code>model1 = ske.ExtraTreesClassifier()
model1.fit(X,Y)
print(model1.feature_importances_)<jupyter_output>[ 0.17934446  0.05316964  0.18672492  0.14219879  0.09248251  0.09302247
  0.07314364  0.0178865   0.03032723  0.11052004  0.02117979]<jupyter_text>The least importatnt features are **Country**, **Sex**, **Race** and **Workclass**  K-fold Cross-ValidationK-fold is the most reliable algorithm for evaluation of any Machine Learning Technique. Is divides the dataset into k random splits and at each iteration it uses one piece to test and rest of the other to train. The process goes on till the algorithm is tested k times. Finally, it returns mean value of scores of each of the k iterations.We are using K-Fold cross-validation with 10 splits of the dataset and a seed value of 7 in order to get consistent results<jupyter_code>seed = 7 
kfold = KFold(n_splits=10, random_state=seed)<jupyter_output><empty_output><jupyter_text>Improving Decision Tree through BaggingBagging reduces the variance of the DesisionTreeClassifier and gives better results. It takes multiple samples of trees from the same training dataset and returns the mean value. The number of trees B is not a critical parameter with bagging; using a very large value of B will not lead to overﬁtting. In practice, we use a value of B suﬃciently large that the error has settled down. Using B = 100 is suﬃcient to achieve good performance in this example.<jupyter_code>base = DecisionTreeClassifier() 

#specifying the number of bagged trees as 100
model2 = ske.BaggingClassifier(base_estimator=base, n_estimators= 100, random_state=seed) 
results2 = cross_val_score(model2, X, Y, cv=kfold) 
print(results2.mean())<jupyter_output>0.818836183122<jupyter_text>Further improving the results through Random ForestsIn bagging, most of the trees will use the strong predictor in the top split. Consequently, all of the bagged trees will look quite similar to each other and results in correlation among trees. Random forests further reduce the variance by *decorrelating* bagged trees. When building these decision trees, each time a split in a tree is considered, a random sample of m predictors is chosen as split candidates from the full set of p predictors. The split is allowed to use only one of those m predictors. A fresh sample of m predictors is taken at each split, and typically we choose m ≈√p—that is, the number of predictors considered at each split is approximately equal to the square root of the total number of predictors. In our case the √11 = 3.31.    Determining the best parameters for Random ForestsThere are certain parameters associated with the *RandomForestsClassifier* class of sklearn. We need to determine the best combination of those parameters in order to get accurate results.Following code displays different parameters of RandomForestsClassifier and their default values.<jupyter_code>rf = ske.RandomForestClassifier(random_state = 42)

# Look at parameters used by our current forest
print('Parameters currently in use:\n')
print(rf.get_params())<jupyter_output>Parameters currently in use:

{'bootstrap': True, 'class_weight': None, 'criterion': 'gini', 'max_depth': None, 'max_features': 'auto', 'max_leaf_nodes': None, 'min_impurity_decrease': 0.0, 'min_impurity_split': None, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 10, 'n_jobs': 1, 'oob_score': False, 'random_state': 42, 'verbose': 0, 'warm_start': False}<jupyter_text>Using Random Search with Cross-ValidationIt performs a randomized search on hyperparameters. The parameters of the estimator used to apply these methods are optimized by a cross-validated search over parameter settings. Each parameter is explained in the code below.<jupyter_code># Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]

# Number of features to consider at every split
max_features = ['auto']

# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)

# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]

# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]

# Method of selecting samples for training each tree
bootstrap = [True, False]

# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}

print(random_grid)
from sklearn.model_selection import RandomizedSearchCV

# Use the random grid to search for best hyperparameters
# First create the base model to tune
rf = ske.RandomForestClassifier(random_state = 42)

# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
# accuracy scoring is used, suitable for classification setting
rf_random = RandomizedSearchCV(estimator=rf, param_distributions=random_grid,
                              n_iter = 100, scoring='accuracy', 
                              cv = 3, verbose=2, random_state=42, n_jobs= -1,
                              return_train_score=True)

# Fit the random search model
rf_random.fit(X, Y);

rf_random.best_params_<jupyter_output>Fitting 3 folds for each of 100 candidates, totalling 300 fits<jupyter_text>Deploying the Random Forests with the best parametes derived from RandomizedSreachCV<jupyter_code># using random selection on 3 features and classification with 100 trees
model3 = ske.RandomForestClassifier(n_estimators= 2000, max_depth = 10, min_samples_split = 10, min_samples_leaf = 1,
                                    bootstrap = True, random_state = seed ) 
results3 = cross_val_score(model3, X, Y, cv=kfold) 

# mean() method return the mean value of the accuracy
# can be converted into percentage by multiplying with 100
print(results3.mean())
print(results3.std())<jupyter_output>0.00669766350404<jupyter_text>Boosting AlgorithmsHere trees are constructed in a sequence with improvement in each step. Boosting Algorithms are usually slower than bagging but they could be more accurate. Let us See!!  Gradient Boosting Gradient boosting (GB) is a machine learning algorithm developed in the late '90s that is still very popular. It produces state-of-the-art results for many commercial (and academic) applications.The aim of gradient boosting is to create (or "train") an ensemble of trees, given that we know how to train a single decision tree. This technique is called boosting because we expect an ensemble to work much better than a single estimator.  Determining the best parameters for Gradient BoostingEarlier we determined the best set of parameters for Random Forests. Similarly, using RandomizedSrearchCV we can obtain the best parameters for Gradient Boosting as well. The best parameters obtained are as follows:{'max_depth': 3,   'min_samples_leaf': 2,   'min_samples_split': 10,   'n_estimators': 200,   'warm_start': True}  Deploying the GradientBoostingClassifier with the best parameters derived from RandomizedSreachCV<jupyter_code>#gradient Boosting
model5 = ske.GradientBoostingClassifier(max_depth = 3, min_samples_leaf = 2, min_samples_split = 10,
                                        warm_start = True, n_estimators= 200, random_state = 42)
results5 = cross_val_score(model5, X, Y, cv=kfold) 
print(results5.mean())<jupyter_output>0.839600214385
{'fraction_non_alphanumeric': 0.07677109793959921, 'fraction_numerical': 0.028224668360146768, 'mean_word_length': 4.8277138157894735, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '608884', 'license': 'MIT', 'n_tokens_mistral': 4326, 'n_tokens_neox': 3927, 'n_words': 1598, 'path': 'AdultData-Shephertz.ipynb', 'repo_name': 'Ktk25/Adultdata---Shephertz-Technologies', 'size': '14172'}