<jupyter_start><jupyter_text>Introduction to Ensemble Methods What are Ensemble MethodsEnsemble methods is a machine learning technique that combines several base models in order to produce one optimal predictive model. This helps us to decrease variance (bagging), bias (boosting), or improve predictions (stacking).Ensemble methods can be divided into two groups:1. Sequential where the base learners are generated sequentially (e.g. AdaBoost).2. Parallel where the base learners are generated in parallel (e.g. Random Forest). Emsemble methods can also be categorized as:1. Homogenous which use a single base learning algorithm to produce homogeneous base learners, i.e. learners of the same type2. Hetrogenous which use heterogeneous learners, i.e. learners of different types*Note: In order for ensemble methods to be more accurate than any of its individual members, the base learners have to be as accurate as possible and as diverse as possible* Terminologies**Bagging:**  Bagging stands for **B**ootstrap **agg**regation. One way to reduce the variance of an estimate is to average together multiple estimates. Variance is the amount that the estimate of the target function will change if different training data was used. Bagging uses bootstrap sampling to obtain the data subsets for training the base learners. For aggregating the outputs of base learners, bagging uses voting for classification and averaging for regression.*Note: The bootstrap sampling method is a statistical technique for estimating quantities about a population by averaging estimates from multiple small data samples. Importantly, samples are constructed by drawing observations from a large data sample one at a time and returning them to the data sample after they have been chosen. This allows a given observation to be included in a given small sample more than once. This approach to sampling is called sampling with replacement.*Imagine we have a dataset with 6 observations:<jupyter_code>dataset = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]<jupyter_output><empty_output><jupyter_text>Now we are going to choose a random observation from the above dataset.<jupyter_code>sample = [0.2]<jupyter_output><empty_output><jupyter_text>This observation is returned to the dataset and we repeat this step 3 more times.<jupyter_code>sample = [0.2, 0.1, 0.5, 0.1]<jupyter_output><empty_output><jupyter_text>We now have our data sample. The example purposefully demonstrates that the same value can appear zero, one or more times in the sample. Here the observation 0.1 appears twice.An estimate can then be calculated on the drawn sample. Those observations not chosen for the sample may be used as out of sample observations.<jupyter_code>oosb = [0.3, 0.4, 0.6]<jupyter_output><empty_output><jupyter_text>In the case of evaluating a machine learning model, the model is fit on the drawn sample and evaluated on the out-of-bag sample. We do not have to implement the bootstrap method manually. The scikit-learn library provides an implementation that will create a single bootstrap sample of a dataset.The resample() scikit-learn function can be used.![Bagging Picture](./bagging.png)<jupyter_code># Standard Libraries required for reading data & perform exploratory analytics
import numpy as np 
import pandas as pd 
import seaborn as sns
import matplotlib.pyplot as plt
train = pd.read_csv('train.csv')
train.shape
train.sample(5)
#  Check  for missing values in the target field which is Survived
print(train.isnull().any())
# Create Titles
train['Title'] = train.Name.str.extract(' ([A-Za-z]+)\.', expand=False)
pd.crosstab(train['Title'], train['Sex'])
# Combine multiple titles to simplify
train['Title'] = train['Title'].replace(['Lady', 'Countess','Capt', 'Col','Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')
train['Title'] = train['Title'].replace('Mlle', 'Miss')
train['Title'] = train['Title'].replace('Ms', 'Miss')
train['Title'] = train['Title'].replace('Mme', 'Mrs')
pd.crosstab(train['Title'], train['Sex'])
# Convert Titles to Numbers
title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5}
train['Title'] = train['Title'].map(title_mapping)
train['Title'] = train['Title'].fillna(0)
pd.crosstab(train['Title'], train['Sex'])
# Dropping Name and Passenger Ids as it will not add any value
train.drop(['Name', 'PassengerId'], axis=1, inplace=True)
train.head()
# Encoding values for sex category
train['Sex'] = train['Sex'].map( {'female': 1, 'male': 0} ).astype(int)
train.head()
# Looking for NaN
print(train.isnull().any())
train.Age.value_counts()
train[train.Age.isnull()]<jupyter_output><empty_output><jupyter_text>We see age with decimal values, for simpicity lets transform age to proper absoluete numbers and also populate the NaN records. We will populate the NaN records using the Median age of the person. As the median age of the person can vary based on his gender and also has dependency on which class he is travelling in.<jupyter_code>train.Pclass.value_counts()
male_age = [0, 0, 0]
female_age = [0, 0, 0]

for j in range(1, 4):
    no_nan_df_male_age = train[(train['Sex'] == 0) & (train['Pclass'] == j)]['Age'].dropna()
    no_nan_df_female_age = train[(train['Sex'] == 1) & (train['Pclass'] == j)]['Age'].dropna()
    male_age[j-1] = no_nan_df_male_age.median()
    female_age[j-1] = no_nan_df_female_age.median()

for j in range(1, 4):
    train.loc[(train.Age.isnull()) & (train.Sex == 0) & (train.Pclass == j),'Age'] = male_age[j-1]
    train.loc[(train.Age.isnull()) & (train.Sex == 1) & (train.Pclass == j),'Age'] = female_age[j-1]

print(train.isnull().any())
# Convert Age from floating point to numbers
train['Age'] = train['Age'].astype(int)
train.sample(5)
train.Age.value_counts()
train['AgeGroup'] = pd.cut(train['Age'], 5)
train[['AgeGroup', 'Survived']].groupby(['AgeGroup'], as_index=False).count().sort_values(by='AgeGroup', ascending=True)<jupyter_output><empty_output><jupyter_text>We can clearly observe that Agegroup 64 to 80 the survival rate is very low<jupyter_code>train.loc[ train['Age'] <= 16, 'Age'] = 0
train.loc[(train['Age'] > 16) & (train['Age'] <= 32), 'Age'] = 1
train.loc[(train['Age'] > 32) & (train['Age'] <= 48), 'Age'] = 2
train.loc[(train['Age'] > 48) & (train['Age'] <= 64), 'Age'] = 3
train.loc[train['Age'] > 64, 'Age'] = 4
train.head()
train.Age.value_counts()
train.drop('AgeGroup', axis=1, inplace=True)
train[['SibSp', 'Parch', 'Survived']].groupby(['SibSp','Parch'], as_index=False).count().sort_values(by='Survived', ascending=False)<jupyter_output><empty_output><jupyter_text>Let's consider a person is alone if SibSp + Parch = 0 otherwise he is with family<jupyter_code>train['IsAlone'] = 0
train['FamilySize'] = train['SibSp'] + train['Parch']
train.loc[train['FamilySize']==0, 'IsAlone'] = 1
train[['IsAlone', 'Survived']].groupby(['IsAlone'], as_index=False).count().sort_values(by='Survived', ascending=False)
train = train.drop(['Parch', 'SibSp', 'FamilySize'], axis=1)
# Let's fill in other NAN records
print(train.isnull().any())
train.Embarked.value_counts()
# Cabin, Embarked neeed to be handled
freq_port = train.Embarked.dropna().mode()[0]
freq_port
train['Embarked'] = train['Embarked'].fillna(freq_port)
train.head()
train.Cabin.value_counts()
# Dropping Cabin column altogether
train.drop('Cabin', axis=1, inplace=True)
train.Fare.value_counts()
#Let's group the fares like Age
train.loc[ train['Fare'] <= 7.91, 'Fare'] = 0
train.loc[(train['Fare'] > 7.91) & (train['Fare'] <= 14.454), 'Fare'] = 1
train.loc[(train['Fare'] > 14.454) & (train['Fare'] <= 31), 'Fare']   = 2
train.loc[ train['Fare'] > 31, 'Fare'] = 3
train['Fare'] = train['Fare'].astype(int)
print(train.isnull().any())
train.head()
# Drop Tikcet column
train.drop('Ticket', axis=1, inplace=True)
# Convert Embarked to Ordinal
train['Embarked'] = train['Embarked'].map( {'S': 0, 'C': 1, 'Q': 2} ).astype(int)
train.head()<jupyter_output><empty_output><jupyter_text>Let's work on the model now<jupyter_code>from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(train.drop("Survived", axis=1), train["Survived"], test_size=0.3, random_state=100)
from sklearn.model_selection import cross_val_score, cross_val_predict
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# print all the scorers of classifier: accuracy score, classification report and confusion matrix
def scores(clf, X_train, y_train, X_test, y_test, train=True):
    if train:
        print("\nTrain Result:\n")
        print("accuracy score: {0:.4f}\n".format(accuracy_score(y_train, clf.predict(X_train))))
        print("Classification Report: \n {}\n".format(classification_report(y_train, clf.predict(X_train))))
        print("Confusion Matrix: \n {}\n".format(confusion_matrix(y_train, clf.predict(X_train))))

        res = cross_val_score(clf, X_train, y_train, cv=10, scoring='accuracy')
        print("Average Accuracy: \t {0:.4f}".format(np.mean(res)))
        print("Accuracy SD: \t\t {0:.4f}".format(np.std(res)))
    else:
        print("\nTest Result:\n")        
        print("accuracy score: {0:.4f}\n".format(accuracy_score(y_test, clf.predict(X_test))))
        print("Classification Report: \n {}\n".format(classification_report(y_test, clf.predict(X_test))))
        print("Confusion Matrix: \n {}\n".format(confusion_matrix(y_test, clf.predict(X_test))))    
# Decision Treee
clf = DecisionTreeClassifier(random_state=100)

clf.fit(X_train, y_train)

scores(clf, X_train, y_train, X_test, y_test, train=True)
scores(clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8060

Classification Report: 
               precision    recall  f1-score   support

           0       0.79      0.91      0.85       159
           1       0.84      0.65      0.73       109

    accuracy                           0.81       268
   macro avg       0.81      0.78      0.79       268
weighted avg       0.81      0.81      0.80       268


Confusion Matrix: 
 [[145  14]
 [ 38  71]]<jupyter_text>So our decision tree has an accuracy of 0.8060Let's try bagging now<jupyter_code>bag_clf = BaggingClassifier(base_estimator=clf, n_estimators=1000,
                            bootstrap=True, oob_score=True, n_jobs=-1,
                            random_state=100)

bag_clf.fit(X_train, y_train)

scores(bag_clf, X_train, y_train, X_test, y_test, train=True)

scores(bag_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8172

Classification Report: 
               precision    recall  f1-score   support

           0       0.81      0.91      0.85       159
           1       0.83      0.69      0.75       109

    accuracy                           0.82       268
   macro avg       0.82      0.80      0.80       268
weighted avg       0.82      0.82      0.81       268


Confusion Matrix: 
 [[144  15]
 [ 34  75]]<jupyter_text>We can see the impact of bagging and the accuracy score now is 0.8172 (vs 0.8060 for decision tree) **Bagging - RandomForest:** Random Forest Models can be thought of as BAGGing, with a slight tweak. When deciding where to split and how to make decisions, BAGGed Decision Trees have the full disposal of features to choose from. Therefore, although the bootstrapped samples may be slightly different, the data is largely going to break off at the same features throughout each model. In contrary, Random Forest models decide where to split based on a random selection of features. Rather than splitting at similar features at each node throughout, Random Forest models implement a level of differentiation because each tree will split based on different features. This level of differentiation provides a greater ensemble to aggregate over, ergo producing a more accurate predictor. Refer to the image for a better understanding.<jupyter_code>from sklearn.ensemble import RandomForestClassifier
rf_clf = RandomForestClassifier(random_state=100)
rf_clf.fit(X_train, y_train)
scores(rf_clf, X_train, y_train, X_test, y_test, train=True)
scores(rf_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8172

Classification Report: 
               precision    recall  f1-score   support

           0       0.81      0.90      0.85       159
           1       0.83      0.70      0.76       109

    accuracy                           0.82       268
   macro avg       0.82      0.80      0.80       268
weighted avg       0.82      0.82      0.81       268


Confusion Matrix: 
 [[143  16]
 [ 33  76]]<jupyter_text>The accuuracy score is 0.8172, same as with bagging. Let's try using GridSearch<jupyter_code>from sklearn.pipeline import Pipeline

from sklearn.model_selection import GridSearchCV
rf_clf = RandomForestClassifier(random_state=100)
grid_params = {"max_depth": [3, None],
               "min_samples_split": [2, 3, 10],
               "min_samples_leaf": [1, 3, 10],
               "bootstrap": [True, False],
               "criterion": ['gini', 'entropy']}
grid_search = GridSearchCV(rf_clf, grid_params,
                           n_jobs=-1, cv=5,
                           verbose=1, scoring='accuracy')
grid_search.fit(X_train, y_train)
grid_search.best_score_
grid_search.best_estimator_.get_params()
scores(grid_search, X_train, y_train, X_test, y_test, train=True)
scores(grid_search, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8172

Classification Report: 
               precision    recall  f1-score   support

           0       0.80      0.93      0.86       159
           1       0.87      0.65      0.74       109

    accuracy                           0.82       268
   macro avg       0.83      0.79      0.80       268
weighted avg       0.82      0.82      0.81       268


Confusion Matrix: 
 [[148  11]
 [ 38  71]]<jupyter_text>We can clearly see GridSearch also provided the same accuracy score, so the default params of Random Forest works better in this case.We tried bagging techniques, lets try the Boosting techniques now. **Boosting:** Boosting refers to a family of algorithms that are able to convert weak learners to strong learners. The main principle of boosting is to fit a sequence of weak learners − models that are only slightly better than random guessing, such as small decision trees − to weighted versions of the data. More weight is given to examples that were misclassified by earlier rounds. The predictions are then combined through a weighted majority vote (classification) or a weighted sum (regression) to produce the final prediction. The principal difference between boosting and the committee methods, such as bagging, is that base learners are trained in sequence on a weighted version of the data.Following are the common types of boosting algorithms:1. **AdaBoost:** Widely used form of boosting algorithm called AdaBoost, which stands for adaptive boosting.2. **GradientBoost:** Utilizes the gradient descent to pinpoint the challenges in the learners’ predictions used previously3. **XGBoost:** Implements decision trees with boosted gradient, enhanced performance, and speed*Note: The goal of any supervised machine learning algorithm is to achieve low bias and low variance. In turn the algorithm should achieve good prediction performance.*![BIAS vs Variance](biasvsvariance.png)<jupyter_code>from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier()

ada_clf.fit(X_train, y_train)

scores(ada_clf, X_train, y_train, X_test, y_test, train=True)
scores(ada_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8097

Classification Report: 
               precision    recall  f1-score   support

           0       0.82      0.87      0.84       159
           1       0.80      0.72      0.75       109

    accuracy                           0.81       268
   macro avg       0.81      0.79      0.80       268
weighted avg       0.81      0.81      0.81       268


Confusion Matrix: 
 [[139  20]
 [ 31  78]]<jupyter_text>We can observe the accuracy score as 0.8097 which is more or less same as that of DecisionTree Classifier(0.8060). Lets try adaboost over Randomforest.<jupyter_code>ada_clf = AdaBoostClassifier(RandomForestClassifier())

ada_clf.fit(X_train, y_train)

scores(ada_clf, X_train, y_train, X_test, y_test, train=True)
scores(ada_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8134

Classification Report: 
               precision    recall  f1-score   support

           0       0.81      0.89      0.85       159
           1       0.82      0.70      0.75       109

    accuracy                           0.81       268
   macro avg       0.81      0.80      0.80       268
weighted avg       0.81      0.81      0.81       268


Confusion Matrix: 
 [[142  17]
 [ 33  76]]<jupyter_text>We can observe that adaboost with Randomforest(0.8134)is also more or less same as Randomforest (0.8172)<jupyter_code>ada_clf = AdaBoostClassifier(base_estimator=RandomForestClassifier())

ada_clf.fit(X_train, y_train)

scores(ada_clf, X_train, y_train, X_test, y_test, train=True)
scores(ada_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8097

Classification Report: 
               precision    recall  f1-score   support

           0       0.81      0.89      0.85       159
           1       0.82      0.69      0.75       109

    accuracy                           0.81       268
   macro avg       0.81      0.79      0.80       268
weighted avg       0.81      0.81      0.81       268


Confusion Matrix: 
 [[142  17]
 [ 34  75]]<jupyter_text>Let's try GradientBoost<jupyter_code>from sklearn.ensemble import GradientBoostingClassifier

gbc_clf = GradientBoostingClassifier()
gbc_clf.fit(X_train, y_train)

scores(gbc_clf, X_train, y_train, X_test, y_test, train=True)
scores(gbc_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8172

Classification Report: 
               precision    recall  f1-score   support

           0       0.80      0.93      0.86       159
           1       0.87      0.65      0.74       109

    accuracy                           0.82       268
   macro avg       0.83      0.79      0.80       268
weighted avg       0.82      0.82      0.81       268


Confusion Matrix: 
 [[148  11]
 [ 38  71]]<jupyter_text>Gradient Boost also yielded same as Randomforest. So for we saw 0.8172 as the highest accuracy score. Let's try XGBoost<jupyter_code>import xgboost as xgb

xgb_clf = xgb.XGBClassifier(max_depth=5, n_estimators=10000, learning_rate=0.3,
                            n_jobs=-1, use_label_encoder=False)
                            
xgb_clf.fit(X_train, y_train)
scores(xgb_clf, X_train, y_train, X_test, y_test, train=True)
scores(xgb_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8209

Classification Report: 
               precision    recall  f1-score   support

           0       0.81      0.91      0.86       159
           1       0.84      0.70      0.76       109

    accuracy                           0.82       268
   macro avg       0.82      0.80      0.81       268
weighted avg       0.82      0.82      0.82       268


Confusion Matrix: 
 [[144  15]
 [ 33  76]]<jupyter_text>XGBoost gave us a marginal improvement taking the score to 0.8209, which is the highest now.Now lets try stacking techniques to combine multiple models and predict. **Stacking:** Stacked Generalization or “Stacking” for short is an ensemble machine learning algorithm. It involves combining the predictions from multiple machine learning models on the same dataset, like bagging and boosting. Stacking answers the following question: "Given multiple machine learning models that are skillful on a problem, but in different ways, how do you choose which model to use (trust)?"The approach to this question is to use another machine learning model that learns when to use or trust each model in the ensemble.* Unlike bagging, in stacking, the models are typically different (e.g. not all decision trees) and fit on the same dataset (e.g. instead of samples of the training dataset).* Unlike boosting, in stacking, a single model is used to learn how to best combine the predictions from the contributing models (e.g. instead of a sequence of models that correct the predictions of prior models).The architecture of a stacking model involves two or more base models, often referred to as level-0 models, and a meta-model that combines the predictions of the base models, referred to as a level-1 model.* **Level-0 Models (Base-Models):** Models fit on the training data and whose predictions are compiled.* **Level-1 Model (Meta-Model):** Model that learns how to best combine the predictions of the base models.<jupyter_code># Let's Recap the Decession Tree and GradientBoost Classifier score
clf = DecisionTreeClassifier()
gbc_clf = GradientBoostingClassifier()
gbc_clf.fit(X_train, y_train)
clf.fit(X_train, y_train)
print("Decision Tree Scores")
scores(clf, X_train, y_train, X_test, y_test, train=False)
print("GradientBoost Scores")
scores(gbc_clf, X_train, y_train, X_test, y_test, train=False)
# Let's try Logistic Regression
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(X_train, y_train)
print("Logistic Regression Scores")
scores(gbc_clf, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Logistic Regression Scores

Test Result:

accuracy score: 0.8172

Classification Report: 
               precision    recall  f1-score   support

           0       0.80      0.93      0.86       159
           1       0.87      0.65      0.74       109

    accuracy                           0.82       268
   macro avg       0.83      0.79      0.80       268
weighted avg       0.82      0.82      0.81       268


Confusion Matrix: 
 [[148  11]
 [ 38  71]]<jupyter_text>We can clearly observe the following accuracy scores:* DecissionTree = 0.8060* GradientBoost = 0.8172* Logistic = 0.8172Now lets stack these models and use Voting Technique to predict.<jupyter_code># importing voting classifer
from sklearn.ensemble import VotingClassifier

model1 = DecisionTreeClassifier()
model2 = GradientBoostingClassifier()
model3 = LogisticRegression()

# Making the final model using voting classifier
final_model = VotingClassifier(estimators=[('clf', model1), ('gbc_clf', model2), ('lr', model3)], voting='hard')

# training all the model on the train dataset
final_model.fit(X_train, y_train)

# Print Scores
scores(final_model, X_train, y_train, X_test, y_test, train=True)
scores(final_model, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8209

Classification Report: 
               precision    recall  f1-score   support

           0       0.80      0.92      0.86       159
           1       0.86      0.67      0.75       109

    accuracy                           0.82       268
   macro avg       0.83      0.80      0.81       268
weighted avg       0.83      0.82      0.82       268


Confusion Matrix: 
 [[147  12]
 [ 36  73]]<jupyter_text>You can observe how marginally the accuracy improved. Now we get the accuracy as 0.8209 which is pretty much same as XGBoost but without using it.Let's use the meta classifier as Logistic Regression and instead of using Voting, we will use the StackingClassifier.<jupyter_code># importing stacking lib
from sklearn.ensemble import StackingClassifier

# putting all base model objects in one list
all_models = [('clf', model1), ('gbc_clf', model2), ('lr', model3)]

# Define Meta Learner Model
meta_model = model3

# Stacking Classifier
final_model = StackingClassifier(estimators=all_models, final_estimator=meta_model, cv=5)

# training all the model on the train dataset
final_model.fit(X_train, y_train)

# Print Scores
scores(final_model, X_train, y_train, X_test, y_test, train=True)
scores(final_model, X_train, y_train, X_test, y_test, train=False)<jupyter_output>Test Result:

accuracy score: 0.8172

Classification Report: 
               precision    recall  f1-score   support

           0       0.80      0.92      0.86       159
           1       0.86      0.66      0.75       109

    accuracy                           0.82       268
   macro avg       0.83      0.79      0.80       268
weighted avg       0.82      0.82      0.81       268


Confusion Matrix: 
 [[147  12]
 [ 37  72]]
{'fraction_non_alphanumeric': 0.09723627240723506, 'fraction_numerical': 0.04536074131024157, 'mean_word_length': 3.2802216834083824, 'pattern_counts': {'":': 10, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 49, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '258438', 'license': 'CC0-1.0', 'n_tokens_mistral': 8428, 'n_tokens_neox': 7588, 'n_words': 2558, 'path': 'aiml/emsemble-methods/intro-to-ensemble.ipynb', 'repo_name': 'saransathy/pycode', 'size': '24713'}